#mainTextBox Two of the most famous unsupervised learning paradigms are principal component analysis (PCA) and competitive learning. PCA linearly projects the input space onto an orthogonal output space of lower dimension while preserving maximum energy of the output. PCA is ideal for reducing the dimension of correlated inputs. Competitive learning is a clustering technique for performing vector quantization.
#subtitleTextBox Creating data clusters
#mainTextBox The data is generated by the Function and Noise components acting together. The Function component produces a square wave of period 2, while the Noise component adds zero mean uniform noise with a 0.15 variance. We have attached a scatter plot probe to the output and set it to accumulate the data of 100 x-y pairs, defined here as one epoch. Run the input for one epoch and observe the effect of the noise on the square wave.
#subtitleTextBox Principal component analysis
#mainTextBox This is the basic architecture for PCA. It consists of a Sangers synapse followed by an axon. Under Sanger's rule, the eigenvectors are ordered (by output energy), and the weights become normalized to one. Oja's rule is also available in NeuroSolutions. Run the simulation for 30 epochs. Observe how the weights find the principal axes, defined by the line that joins the clusters and its perpendicular.
#subtitleTextBox Annealing the learning rate
#mainTextBox During the last simulation you may have noticed how fast the weights converged but that the major axis's weight track jumped around. This is because the learning rate was set too high. Ideally, the learning rate for PCA should be annealed. This allows rapid convergence of the high-energy components at the beginning of training and fine-tuning of the lower energy components toward the end of training. Therefore, we have added a scheduler to the step size. Repeat the experiment.
#subtitleTextBox Competitive learning
#mainTextBox The strategy of standard competitive learning is to find the centers of a pre-defined number of clusters. The number of output PE's determines the number of clusters the system is trying to identify. We have replaced the Sangers synapse with a Standard competitive synapse. Run the experiment and watch what happens.
#subtitleTextBox Non-symmetric data clusters
#mainTextBox The standard competitive rule uses the dot product metric. A shortcoming of this rule is that the clusters must be symmetric about the origin. We have moved the clusters so that they are unsymmetric about the origin. Run the network and verify that the weights can't find the data clusters.
#subtitleTextBox Superfluous PEs
#mainTextBox Another problem with standard competitive learning is that it is necessary to know the number of data clusters apriori. Enter a value for the number of output PEs greater than the number of clusters, and re-train the network. Observe how the extra PEs are superfluous.
#subtitleTextBox Conscience
#mainTextBox One way around these problems is to use a conscience. A conscience keeps track of how frequently each neuron wins and inhibits neurons from winning if they have won too often in the recent past. We have replaced the Standard competitive synapse with a Conscience competitive synapse. Re-run the experiment and observe that conscience solves both the problem of non-symmetric data and superfluous PEs.
#mainTextBox If class labels are available, then a supervised clustering procedure can be used. With LVQ, the winning PE is moved toward the data point if it is the correct class, and away from it otherwise. The class labels (0 or 1) are input at the "forced" access point of the Standard synapse. Run the experiment and observe that the same color PE always goes to the same cluster. Change the class definition, re-run the experiment, and observe that the same PE now goes to the other cluster.
#subtitleTextBox Summary
#mainTextBox You have seen how NeuroSolutions implements two of the most popular unsupervised learning paradigms: principal component analysis and competitive learning.
